Attack-Test and Verification Systems, steps towards verifiable Anomaly Detection

Botnet, network malware and anomaly detection algorithms are hard to evaluate and compare against each other due to different data sets. In some cases overspecialization on known malware gives high detection rates due to unknown artifacts in the training data set. This may lead to new malware being unnoticed on a network, because the detection algorithm has not been optimized for this case. Our proposal is a new and work-in-progress approach to generate parametricized and randomized testing data sets on the fly. We plan to couple this with the an automatic verification system to assess the quality of detection algorithms without internal knowledge of their working. We hope to encourage discussion to enhance the draft of our idea and especially to go into more detail on our work in progress.